Skip to content

WIP WIP vulkan schema generator and struct decode/encode action implementation - #3250

Open
jzulauf-lunarg wants to merge 9 commits into
LunarG:devfrom
jzulauf-lunarg:jzulauf-vulkan-schema-generator
Open

jzulauf-lunarg wants to merge 9 commits into
LunarG:devfrom
jzulauf-lunarg:jzulauf-vulkan-schema-generator

Conversation

@jzulauf-lunarg

@jzulauf-lunarg jzulauf-lunarg commented Sep 6, 2026 •

Copy link
Copy Markdown
Contributor

All struct decode through a central action functor
All struct encode through a central action functor

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 6, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-06T02:24:37.917561Z eb105bf PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@jzulauf-lunarg
jzulauf-lunarg force-pushed the jzulauf-vulkan-schema-generator branch from a49eebf to d4e771f Compare September 15, 2026 02:27
@jzulauf-lunarg jzulauf-lunarg changed the title WIP WIP WIP vulkan schema generator and decode action implementation WIP WIP vulkan schema generator and struct decode/encode action implementation Sep 19, 2026
@jzulauf-lunarg
jzulauf-lunarg force-pushed the jzulauf-vulkan-schema-generator branch from c766951 to 3315c60 Compare September 29, 2026 20:47
@jzulauf-lunarg jzulauf-lunarg added the approved-to-run-ci Can run CI check on internal LunarG machines label Sep 29, 2026
@jzulauf-lunarg
jzulauf-lunarg force-pushed the jzulauf-vulkan-schema-generator branch from 3315c60 to e4e4a38 Compare September 29, 2026 22:08
@jzulauf-lunarg

Copy link
Copy Markdown
Contributor Author

All Generated Decode and Encode bodies removed. Schema traits, means we're still at an increase in KLOC.

Next step is removing explicit sType sieves. LOC will start to decrease.

@jzulauf-lunarg
jzulauf-lunarg force-pushed the jzulauf-vulkan-schema-generator branch from 0a28bde to 7e4c49d Compare October 5, 2026 17:48
jzulauf-lunarg and others added 9 commits October 5, 2026 14:56
…ery structure decoder

Pilot of the Generic Field Schema and Action Model (Design-documents,
consumer-graph/decode_generic_visitor_action_model.md, v7.0) for one
operation family: decoding a Vulkan structure from a capture stream.

What is added

  framework/format/format.h gains a vocabulary of logical kinds, one tag
  per encode-type typedef, each carrying the wire type it is recorded
  as. A kind is not a type: VkDeviceSize and uint64_t are the same bytes
  and the kind is what tells them apart.

  framework/schema/ is a new header-only, API-agnostic layer: field
  shapes, the MemberPointer trait and its Addressable/HasMember
  concepts, GetRef/Get/Set, the kind-and-shape concepts an Action
  selects on, Schema<ApiElement>::Fields, WalkFields, the command
  return partition and InvokeFromFields. framework/util/type_list.h
  supplies the TypeList facility it composes.

  A new generator, vulkan_schema_generator.py, emits six files from
  vk.xml: API type descriptors, command tags, Field descriptors and
  Schema specializations; the decoded-representation traits in both
  directions (ApiElementTraits and its inverse ApiElementFor); three
  member-trait partitions for native structures, decoded wrappers and
  decoded command args; and a compile-time checks file. Curated
  knowledge the registry lacks is small and named: four refined kinds
  (size_t, VkDeviceSize, VkDeviceAddress, VkSampleMask) and three
  shared descriptors (GenericHandle with its selector field,
  ExternalObject, OpaqueBytes).

  framework/decode/vulkan_decode_action.h is the one hand-written
  Action: twelve constrained Apply overloads covering every field shape
  the registry produces. Every write to the decoded value goes through
  Set; GetRef appears only where a decoder needs a member's address.
  vulkan_decode_struct_impl.h defines DecodeStruct as one constrained
  function template around WalkFields. ValueDecoder and PointerDecoder
  gain a Decode<Kind> template so the Action names kinds, not widths.

What is replaced

  All 1,343 non-union, non-blacklisted Vulkan structures decode through
  the field walk. generated_vulkan_struct_decoders.cpp drops 26,000
  lines of procedural bodies for one explicit instantiation per
  structure; one hand-written body (VkBaseOutStructure) remains and is
  the single entry of the NonSchemaDrivenStructs exclusion list. The
  forward header declares one constrained template in place of one
  prototype per structure. Callers are unchanged: every call still
  names a Decoded_T*, and a hand-written non-template overload still
  wins. Only that translation unit compiles the walk; the member-trait
  partitions reach no other target.

Verification

  MSVC Debug and GCC 13.3 Debug build clean; 281 assertions in 19 test
  cases pass on both. test_vulkan_schema.cpp round-trips one structure
  per Apply overload against hand-encoded bytes. The checks file holds
  7,309 static_asserts: every structure has a schema and no return
  field, every command has one return field, both trait directions
  round-trip, every descriptor's element type equals its wrapper's
  struct_type, every scalar element is as wide as its kind's wire type,
  and every refined kind records the same bytes as the primitive the
  encoder resolves it to. Regeneration reproduces the tree byte for
  byte. Before the procedural bodies were removed, the two decoders
  were compared field for field on the same buffers.

Not in this change

  No semantic metadata is emitted. Commands have schemas, tags, traits
  and member traits but no decode Action; two pointer-shaped identifier
  overloads will be needed when they do. No capture_wrapper_type, no
  formal API-signature storage, no dispatch or lifecycle traits. Unions
  keep their hand-written decoders and are reached by legacy descent.
  No CI check yet enforces the private-include boundary. The
  capture-file smoke run and encoded-byte comparison the design's
  proof gates call for have not been recorded.

Known latent generator limits, none reachable from the current
registry: a two-dimensional static array with a Count sibling would
emit an invalid extent; a pointer-returning command would emit an
invalid descriptor name; base-header hierarchies (parentstruct) are
excluded by is_schema_driven rather than modelled.

The 26-commit history this collapses is preserved at tag
jzulauf-vulkan-schema-generator-presquash.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tructure encoder, and no struct body is generated on either side

The Vulkan structure encoders join the decoders on the generated field
schema. EncodeStructAction (framework/encode/vulkan_encode_action.h) walks
a structure's Field descriptors and encodes each by shape and kind through
an adapter over ParameterEncoder; generated_vulkan_struct_encoders.cpp is
1,347 explicit instantiations and no body. The decoders reach the same
state: the one remaining generated decode body, the base-out parameter of
the ARM data-graph query, now decodes through the pNext sType sieve into
TypedStructDecoder, and its replay allocation is five lines over the
existing pNext output initializer. EncodeStruct<Struct> and
DecodeStruct<Wrapper> are plain function templates; there is no list of
structures the schema does not drive, since there are none, and a
hand-written non-template overload is preferred by overload resolution
where one exists. The generated encoders header includes the template
header, so a caller may include either, as on dev.

Schema
- one generated family split into types, fields and schema, namespaced
  API first, genre second (schema::vulkan::{api_types, fields, commands})
- shapes: Value, Pointer, Array (pointer to a counted run), PointerArray
  (counted run of pointers each to one element), StaticArray,
  ExtensionChain, VoidReturn; concepts <Shape>ShapeField, Any<..>ShapeField,
  <Kind><Shape>Field; kind families are format's to define and the field
  layer lifts them, one predicate per kind
- static array extents recorded on the Field from the registry len; the
  count sibling of a counted static array reaches the descriptor through
  the registry's len attribute (dev PR 3265)
- counts as StoreValue operands: FieldValue<Sibling>, FieldValue<Sibling,
  Member> through a pointer sibling, Constant<V>, and prefix Sum, Product,
  Quotient for the registry's altlen expressions; Get returns
  decltype(auto), Getter is gone, and so are 450 forward declarations
- a decoded wrapper names its own API element

Encode
- every idiom the census found has an adapter entry: scalar values and
  addresses, wrapped and generic handles, text (pointer, fixed extent
  bounded by the extent, and runs), fixed-extent arrays of one or two
  dimensions counted or whole, pointers to one scalar, counted runs, the
  extension chain from the Field's registry fact, opaque bytes, and the
  kind-tag entry points that carry the named entry points' omit flags
- a counted static array is clamped by ParameterEncoder::
  ClampStaticArrayLength, one implementation shared with dev's generated
  callers, a template on the count's type: negative to 0, above capacity
  to capacity, both reported
- the CaptureWrapperFor rows (52, one per handle descriptor) and the
  DescriptorFor rows (1,347) are generated schema-family targets;
  encode/vulkan_encode_descriptor_for.h holds the primary template and
  the row macro
- the scalar-and-address value entry takes the member's declared type,
  asserting it is the element type, an address, or a same-width integral
  typedef (const void*, const CAMetalLayer*, DWORD members)

Proof
- a transitional oracle suite snapshotted all 1,343 procedural bodies and
  compared bytes against the inverted library, 1,343 of 1,343 on MSVC, GCC
  and clang with libc++; nothing became a canary and the suite is retired;
  its schema-driven filler stays as framework/test/schema_fill.h
- capture gate against a build of the merge base: 19 traces on an NV host
  and three on AMD, layer, converter and replay grabs identical block for
  block, state snapshots included
- framework test: 506 assertions in 44 cases on MSVC, GCC and clang,
  dev's static-array cases included; Android builds

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… VisitAt

Enumerants<Enum>::entries carries each enum's values and names in registry
order, generated into generated_vulkan_schema_enumerants.h; util::NameOf
looks a value up through a KeyIndex. KeyIndex is a sorted constexpr index
over an array, and VisitAt calls a visitor's operator()<Element> for the
element at a position in a TypeList, through a table of invokers.

The enum-to-string and enum-to-JSON bodies are dev's, unchanged: they stay
until their generation is replaced. The enumerant test checks that ToString
and to_json name every enumerant as Enumerants does.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… the dispatch pilot

The generated 3,522-line switch and its generator go. In their place:
structure_type on every typed structure descriptor (1,144, from the
registry's values attribute), a descriptor for the 20 typed structures the
schema excludes, and a generated catalog, catalog::structures (1,367) and
catalog::extensible_structures (1,144), both sizes asserted by the checks.
schema::StructureTypeIndex<List> is a static Find/End over a KeyIndex keyed
by structure_type; util::IndexPosition<Index> is what Find returns and
Visit takes, deducing the list from the index. The hand-written
vulkan_pnext_struct_encoder.cpp keeps the loader skip and DF-2's guarded
hop, and dispatches a recognized node through one Visit to EncodeStructPtr.
Every structure_type is checked against GetSType, which stays this turn.

WIP: three fold expressions over the catalog exceed clang's nesting limit
(two static_asserts commented out, one pack constraint loosened to
typename...); the flat forms and a catalog-sized ForEachType are the next
turn's first item (findings section 6). The pNext test samples twelve
structures until then. The framework test gets /bigobj as encode and
decode already have.

Gates: MSVC, GCC, clang with libc++, Android; framework 4,394/46 (pnext 30),
util 7,386/38 (Linux 7,321/32). Encoder TU: 11-15 s per toolchain.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
End-of-week snapshot, MSVC only. The three fold checks disabled in 5edc3fb
return in flat form: a bool array and a constexpr count, no fold. TypeListAt
is Mp11's indexer (one base per element, picked by overload); the filter
builds a positions array, turns it into an index_sequence once
(PositionSequence), and expands over it, so no subscript is evaluated inside
a template argument list. TypeListKeep, TypeListDrop and TypeListCountIf take
the predicate as a value, a captureless lambda with a template call operator,
the shape TypeListForEach's visitors already have; TypeListDrop is
TypeListKeep over Not. IsReturnField becomes kIsReturnField; ReturnMatches is
ReturnFields; TypeAt and ForEachType are TypeListAt and TypeListForEach, so
every list algorithm is spelled TypeList<Op>.

StructureTypeIndex loses its concept constraint so the diagnostic for a
descriptor without structure_type is reachable: StructureTypesOf asserts the
count, and only on failure instantiates the filtered list to name the
offenders. The pNext test walks the whole catalog again; the 300-wide list in
test_key_index.cpp exercises every TypeList algorithm past clang's fold limit.

Not yet run: GCC, clang with libc++, Release, Android; the failure-message
probe. MSVC Debug: util 7,688/39, framework 6,658/46 (pnext 2,294).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r goes

util::GetSType<T>() is one template that returns the structure_type of T's API
type descriptor, found through schema::binding::DescriptorFor. Its name,
namespace and VkStructureType return are unchanged; two caller headers change
one include line each.

DescriptorFor moves from encode to schema::binding, the category of traits
that join a schema entity to a C++ type existing without the schema. The rows
are generated into generated_vulkan_schema_binding_descriptor_for.h, one per
structure descriptor (1,367), where encode's rows covered only the described
structures (1,347). Encode's static_assert names the binding.

Removed: generated_vulkan_stype_util.h (1,201 lines), its generator, its
registrations, and the 1,144 agreement asserts in the schema checks, which
would now compare a value with itself. The shared Khronos struct-type base
generator stays for OpenXR's type util.

Net -2,339 lines: generated -2,324, generator Python -68, hand-written +53.
Gates: MSVC Debug util 7,688/39, framework 6,658/46 (pnext 2,294); GCC Debug
and clang with libc++ Debug util 7,623/33, framework 6,658/46 (pnext 2,294);
Android layer and replay build. Release not run.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…d bitvalues

Follows the checkpoint a30d54d and completes the pass; the two squash later.

Generated to-string bodies are gone. generated_vulkan_enum_to_string.cpp is
490 one-line explicit specializations of util/to_string.h's primaries, each
a lookup over the type's enumerant table; the header keeps its 490
declarations, so callers are unchanged and dev's generic formatters
(ArrayToString, BitmaskToString) still find them, which overloads declared
in another header would not have. The 64-bit flag-bits types, typedefs of
VkFlags64, are keyed on the API type descriptor through two templates
declared in the generated header, defined in util/vulkan_enum_to_string_impl.h
(included only by the .cpp) and instantiated there 32 times; dev's 32 named
functions go and the util test respells. The to-string generators join the
schema family.

Schema: every descriptor carries name, its element type's spelling. An
enumerated type's descriptor names its table (enumerants); the tables hold
entries only, each an Enumerant<Descriptor>. A bits enum's descriptor carries
bitmask, the type its values compose into, from the <enums> group's bitmask
qualifier and bitwidth; a mask typedef's descriptor carries bitvalues, its
bits descriptor, from the typedef's requires/bitvalues. One direction each,
no boolean. is_enumerated replaces has_enumerants: every enum type gets a
table, empty or not. catalog is its own generated header. Checks: bitmask
iff the name says FlagBits; each typedef's element_type is its bits' bitmask;
nothing binds uint32_t or uint64_t.

Enum to JSON is dev's, by decision, until the JSON consumer is replaced.

Gates, fresh binaries: MSVC Debug util 7,691/40, framework 6,658/46;
GCC Debug, clang with libc++ Debug, GCC Release -Werror util 7,626/34,
framework 6,658/46; Android layer and replay build.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…tates its derived lists as relations

graphics/vulkan_struct_deep_copy.cpp (John) replaces the generated
vulkan_struct_deep_copy_stype switch, 1,136 cases: StructureTypeIndex over
catalog::deep_copyable_structures and one Visit into the per-structure
vulkan_struct_deep_copy. The generated switch, its generator and its
registrations are gone.

The catalog now emits only the master list and the curated data; every other
entry is a relation evaluated in C++: extensible_structures is
TypeListKeep(structures{}, kHasStructureType), with kHasStructureType beside
the concept in schema.h; deep_copy_exclusions is the structures-deep-copy key
of blacklists.json, which the deep-copy body generator also reads in place of
its literal; deep_copyable_structures is TypeListExclude of the two. The
checks file asserts each derived size against the generator's count: 9
exclusions, 1,136 deep-copyable. TypeListExclude joins type_list.h with a
300-wide test line.

MSVC Debug: util 7,691/40, framework 6,658/46. The sieve is linked and not yet
exercised by a test; its callers are encode's vulkan_track_struct.h and the
pNext hops inside the generated deep-copy bodies.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@jzulauf-lunarg
jzulauf-lunarg force-pushed the jzulauf-vulkan-schema-generator branch from 221397d to 1078582 Compare October 5, 2026 22:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved-to-run-ci Can run CI check on internal LunarG machines

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant